Adaptive Camera And Illuminator Eyetracker

ABSTRACT

An eye tracker includes at least one illuminator for illuminating an eye, at least two cameras for imaging the eye and a controller. The configuration of the reference illuminator and cameras is such that, at least one camera is coaxial with a reference illuminator and at least one camera is non-coaxial with a reference illuminator. The controller selects one of the cameras to be active to increase an image quality metric and avoid obscuring objects. The eye tracker is operable in a dual-camera mode to improve accuracy.

FIELD OF THE INVENTION

The present invention relates to a device for illuminating and imagingan eye. More precisely, the invention provides an adaptivepupil-centre-corneal-reflection (PCCR) eye tracking system comprisingmultiple cameras.

BACKGROUND OF THE INVENTION

A PCCR-based approach to determining the gaze of an eye may use an imageof the eye either in its bright-pupil condition (a retinalretro-reflection complements the iris image) or dark-pupil condition (acorneo-scleral reflection complements the iris image). In a givensituation, the respective images may be of different quality, and it mayeven be impossible to carry out an uninterrupted gaze tracking based onjust one of these imaging modes. Therefore, to be able to choose theoptimal mode, some available eye trackers comprise double referenceilluminators for creating the reflections. A first referenceilluminator, for use in imaging in the bright-pupil mode, is thenarranged coaxially with the optic axis of a camera (image sensor),whereas a second reference illuminator, for use in the dark-pupil mode,is arranged off the camera axis. Such a reference illuminator may be acompound light source arranged round the camera objective in aconcentric ring; cf. FIG. 3 in Applicant's patent SE 524003.

It is known in the art (see, e.g., the paper General Theory of RemoteGaze Estimation Using the Pupil Center and Corneal Reflections by E. D.Guestrin and M. Eizenmann, IEEE Transactions on Biomedical Engineering,Vol. 53, No. 6, pp. 1124-1133 (June 2006), included herein by reference)that the eye's position and orientation, at a given point in time,cannot be unambiguously determined unless the locations of two distinctcorneal reflection (or glints, or first Purkinje reflections) can beextracted from one image of the eye or from several, simultaneousimages. If two reference illuminators are used simultaneously, however,coexisting glints will mutually blur the measurements by reflections andthe like. If the reference illuminators are used alternately (e.g., bytime interlacing), then a small time delay will necessarily separate thetwo images, to the detriment of the accuracy, particularly if the delayfalls in the duration of a saccade. The delay also makes the eyetracking slower. A similar drawback becomes noticeable if thebright-pupil image is used for providing an initial guess in the processof finding the location of the pupil centre in the dark-pupil image.This is practiced in the art, as described, notably, in patentapplication US 2004/0005083. Since the two images cannot be acquiredsimultaneously, such initial guess is sometimes of little avail.

As many of those skilled in the art will acknowledge, the accuracy ofeye tracking is highly dependent on the resolution of the camera usedfor imaging the eye with the glints. Indeed, the virtual image of thereference illuminator formed by reflection in the cornea is shrunk by afactor 100 or more (assuming a corneal focal length of 4 mm and anilluminator-to-eye distance of at least 400 mm). On the other hand, toavoid serious round-off errors, the image of the reference illuminatorshould occupy a region of at least, say, ten camera pixels. Hence, foran eye tracker to be useful, a reasonably high performance is requiredfrom the camera, which therefore defines a least possible price of theproduct.

Conventional eye trackers generally perform optimally if the studiedperson does not move during a measurement session. Particularly annoyingare head movements that change the angle between the head and the cameraof the eye tracker, because this may introduce obscuring objects intothe line of sight from the reference illuminator to the eye or into theline from the eye to the camera. Notably, spectacle frames, eyelashes,eyebrows, nose and protruding brow bones may cause problems of thiskind.

It is probably similar considerations that have led to the widespreaduse of ring-shaped reference illuminators in eye trackers.Conventionally there is a larger ring for providing off-axisillumination and a smaller ring arranged around the circumference of thecamera objective to be as coaxial as possible. By surrounding all sidesof the camera objective with luminous points, the risk of having thetracked eye obscured is decreased. However, a ring-shaped illuminator isimaged in the cornea as an inhomogeneous spot having lower luminancethan a solid light source would. This is detrimental to image contrastand makes it more difficult to find the location of the reflection ofthe light source. The problem is most severe in the case of the coaxial,smaller illuminator, which is further shrunk by reflection in the convexcornea, as seen above.

In view of the above shortcomings associated with available eyetrackers, there appears to be a need for improved eye-tracking devicesas regards accuracy, speed, reliability and cost efficiency.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a device for eyeillumination and eye imaging, in a manner suitable for subsequentextraction of gaze-point data from eye images obtained by the device.

In accordance with a first aspect of the invention, an eye tracker isprovided which comprises at least one illuminator for illuminating aneye, at least two cameras (or other image sensors) for imaging the eyeand a controller. The configuration of the reference illuminator(s) andcameras is such that, firstly, at least one camera is coaxial with areference illuminator (bright-pupil imaging mode) and, secondly, atleast one camera is non-coaxial with a reference illuminator (dark-pupilimaging mode). The controller is adapted to select at least one of thecameras to be active. The camera selection, which is performedrepeatedly, is based on an image quality metric which is a function ofat least one image quality factor. If one active camera is to beselected, the one which yields the image with the best quality metric ischosen.

In comparison with available devices, an eye tracker according to theinvention is less vulnerable to sight-obscuring objects, such aseyelashes and eyebrows, because of the higher probability of one of thetwo cameras being unhindered. If the line of sight of one camera becomesobscured, the quality-metric value of its image will drop accordinglyand the other camera will be considered for activation. Because anilluminator usually dissipates more power than a camera, the inventionalso represents an energy-economic advantage over available eyetrackers.

The invention can be advantageously embodied as an eye trackercomprising two reference illuminators and two cameras, so that fourcombinations of one illuminator and one camera are possible. One or twocombinations will relate to the bright-pupil mode and two or threecombinations to the dark-pupil mode (for, as noted above, at least onecamera is coaxial with a reference illuminator and at least one isnon-coaxial with a reference illuminator). Thus, at least in thedark-pupil mode, if one line of sight becomes obscured, the eye trackercan continue tracking using a different combination in the same mode,which facilitates subsequent image processing such as pupil finding.This makes operation of the eye tracker more reliable.

In an advantageous variation to the previous embodiment, each camera isassociated with a coaxial, substantially point-shaped illuminator.Necessarily, this enables two bright-pupil combinations of one cameraand one illuminator. A benefit of the point-shapedness of the referenceilluminator is that the corneal reflection of the reference illuminatoris more likely to appear as a small solid spot with good contrastagainst the background, so that the location of the reflection can beaccurately determined. Because two different camera-illuminatorcombinations are available, it is not very likely that both are subjectto obscuration (which is one of the motives of the ring shape of on-axisreference illuminators of prior art). Generally, a reflection of areference illuminator covers a spot of several pixels in the cameraimage, and the location of the reflection, as used herein, refers to thecentre of the spot in some suitable sense, as will be detailed below.

Preferably, the eye tracker is operable in a dual-camera mode, in whichboth cameras are active, which permits truly simultaneous acquisition ofimages in different illumination conditions. As an economic advantage ofthis, the resolution requirement on each camera can be relaxed. Since,for instance, the location of a corneal reflection can be determinedmore accurately when it is imaged by two cameras in distinct positions,the effective accuracy can be retained even though two simpler camerasare used. Assuming the eye tracker is adapted for distances to theviewer in the range up to 1 m, the cameras should be separated by atleast 70 mm, i.e., at least 4.0 degrees of arc, if both bright-pupil anddark-pupil imaging is desired. If one available imaging mode isconsidered sufficient, then the cameras may be arranged closer, such as50 mm apart.

Any embodiment of the invention may advantageously include operabilityin an evaluation mode. The evaluation mode has the purpose of assessingthe quality and usefulness of a large number of availablecamera-illuminator combinations. The evaluation mode entails activatinga plurality of cameras, while the reference illuminators aresequentially scanned. Alternatively, all cameras are activated duringthe scan. This expedites the gathering of images for evaluation.

In order not to distract a viewer's attention, the referenceilluminators used in any embodiments of the invention are preferablyadapted to emit light that is not visible to the human eye. It isadvantageous to use light in a wavelength range adjacent to the visiblespectrum—thus in the infrared or ultraviolet range—because it may thenbe possible to use imaging devices for visible light with only minoradaptations. However, it is known that exposure to ultraviolet radiationmay be harmful to the human body, so that infrared or near-infraredlight is the preferred choice.

As regards the processing of images collected by an eye trackeraccording to the invention, it is advantageous to use a computationalmodel that includes an aspherical geometric model of the cornea.Preferably, to reflect widely recognised optometric facts, anellipsoidal cornea model is used. The eye tracker is adapted todetermine an orientation of the eye based on the locations of thecorneoscleral reflections of the reference illuminators. The gazedirection of the eye is easily determined once the orientation is known.Because a general ellipsoidal surface is not rotationally symmetric, itmay not be necessary for the eye tracker to determine a pupil-centrelocation and take this into account. While the ellipsoidal shape iscommon to the majority of persons, the model may need to be fine-tunedaccording to individual variations in a calibration procedure beforemeasurements are started. The tuneable parameters may include the radiusof corneal curvature at the pupil centre and the corneal eccentricity.

In accordance with a second aspect of the invention, there is provided amethod for selecting a combination of an active reference illuminatorfrom a plurality of reference illuminators and an active camera from aplurality of cameras. Each camera is adapted to image at least one eyewhen illuminated by one or more reference illuminators. At least onecombination is adapted for imaging in the bright-pupil mode (the cameraand the reference illuminator are coaxial) and at least one combinationis adapted for imaging in the dark-pupil mode (they are non-coaxial).The method includes the following steps, to be performed in this order:

An image quality metric, which depends on at least one image qualityfactor, is defined.

Two or more eye images are acquired, at least one in the bright-pupilmode and at least one in the dark-pupil mode. The image quality metricis evaluated for each of the images, and the imaging mode which providesthe greatest value of the quality metric is selected.

Eye images are acquired using available combinations of an active cameraand an active illuminator corresponding to the selected imaging mode,and the image quality metric is evaluated for the images. Preferably,each available camera is included in at least one of the combinationsused for acquiring these eye images. That camera which provides thegreatest value of the quality metric is selected as active camera.

Once an active camera has been selected, an active reference illuminatorremains to be selected. Only a reference illuminator which, incombination with the selected active camera, provides imaging in theselected imaging mode can be chosen. The selection of an activereference illuminator is effected on the basis of the centricity of itscorneo-scleral reflection (first Purkinje reflection): the illuminatorwhich provides the most centric reflection is selected.

This concludes the initial selection of a combination of an activecamera and an active reference illuminator. If the referenceilluminators and cameras are provided in an eye tracker operable in anevaluation mode, as set forth above, then advantageously the eye imagesfor which the image quality metric is evaluated are acquired in thismode.

In an advantageous embodiment of the invention, the method for selectinga camera and an illuminator can be complemented with further steps forcontinually reassessing the selection in an economic and efficientmanner. After the above steps have been completed, it is establishedwhether the image quality metric obtained using the selected combinationis above or below a predetermined threshold. If it is found to be abovethe threshold, then step iv) is repeated, that is, it is checked whetherthe selected reference illuminator actually provides the most centriccorneo-scleral reflection or whether switching to some other referenceilluminator can improve the centricity. If the image quality factor isfound to be below the threshold, the camera selection is revised byrepeating step iii). After this, evidently, step iv) has to be repeated.If the image quality metric is still below the threshold even thoughsteps iii) and iv) have been repeated, then the choice of imaging modeis revised by repeating steps ii), iii) and iv).

It is noted that step iv) does not necessarily imply acquiring a set oftest images, in which the centricity of the corneoscleral reflection isevaluated. Indeed, since the spatial configuration of the availablereference illuminators is usually known a priori, the switching betweendifferent active reference illuminators may be effected based on merelythe actual position of the corneo-scleral reflection in the imagecurrently used for eye tracking. E.g., in a situation where the testsubject looks to the left, so that the corneo-scleral reflectionapproaches a right boundary of the cornea, then a reference illuminatorlocated further to the left should be selected instead of the presentone. Likewise, if the camera positions are known beforehand, thenguidance can be obtained in the camera switching based on the latest eyeposition and gaze direction. Thus, instead of acquiring test images byeach available camera, the cameras most likely to have a better viewingangle are evaluated for selection.

By virtue of its hierarchic nature, the proposed method for updating theselection of an camera-illuminator combination is economical in so faras it limits the number of evaluations of the image quality metric. In atypical computer implementation of the method, this number is likely toinfluence the computational complexity. The switching between referenceilluminators (step iv)) does not require any evaluation of the qualitymetric. The proposed method also minimises the number of times a camerais temporarily taken out of duty to acquire test images, whichinterrupts the gaze tracking. This may occur, for instance, when theselection of imaging mode is reassessed (step ii)) by acquiring a testimage using the currently active camera and a currently inactiveilluminator (one corresponding to the other imaging mode than thatcurrently selected). As regards the reassessment of the camera selection(step iii)), the test images on which the decision is based may beacquired in a dual camera mode, which means that the gaze tracking canbe pursued without interruption.

In accordance with a third aspect of the invention, there is provided acomputer-program product for causing a general-purpose computer toperform the method for selecting a combination of an active referenceilluminator from a plurality of reference illuminators and an activecamera from a plurality of cameras, as set forth above.

These and other aspects of the invention will be apparent from andelucidated with reference to the embodiments described hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will now be described withreference to the accompanying drawings, on which:

FIG. 1 shows a combined camera and illuminator arrangement in accordancewith an embodiment of the invention;

FIG. 2 shows a combined camera and illuminator arrangement in accordancewith another embodiment of the invention;

FIG. 3 shows a combined camera and illuminator arrangement in accordancewith a further embodiment of the invention;

FIG. 4 is a diagrammatic cross-sectional view of the cornea;

FIG. 5 is a diagrammatic perspective drawing showing an array ofreference illuminators, their corneoscleral reflection and a cameradevice adapted to image the eye with said reflection;

FIG. 6 is a flowchart of the method for selecting a combination of acamera and a reference illuminator according to an embodiment of theinvention; and

FIG. 7 is an illustration of a decision tree associated with the methodfor selecting a combination of a camera and a reference illuminator whenapplied to the combined camera and illuminator arrangement of FIG. 2.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

I. Eye Tracker Comprising One Reference Illuminator

FIG. 1 shows a combined camera and illuminator arrangement 100. Thearrangement 100 may be part of an eye tracker, in the sense that it iscontrollable by an eye tracker, and may even be embodied in the samephysical unit as such device. It is also envisioned that an eye trackingsystem may comprise a processor and the arrangement 100 of FIG. 1.

The arrangement 100 comprises a reference illuminator 120 and a firstcamera 110 provided coaxially with the reference illuminator 120 in thesense that the optic axes of the two devices are parallel and thedistance between them is small in relation to the overall length scale.The reference illuminator 120, which preferably is adapted to emit(near)infrared light from a point-shaped aperture, and the first camera110 are arranged so closely to one another that it is possible to imagethe retinal retro-reflection (bright-pupil effect) of the referenceilluminator 120. In some conditions, this may provide an eye image inwhich the pupil circumference is resolved with high accuracy. Thearrangement 100 further comprises a second camera 112 arranged at suchdistance away from the reference illuminator 120 that the retinalretro-reflection is not visible. In other words, the second camera 112is adapted to image the eye in its dark-pupil condition. The referenceilluminator 120 of the arrangement 100 is located between the cameras110, 112, but may also, in an alternative embodiment, be located to theright or left of the cameras 110, 112.

When the arrangement 100 is used in eye tracking, the referenceilluminator 120 and at least one of the cameras are active. (Two activecameras may be required in an initial regime of the eye tracking fordetermining the spatial position of the eye; in eye trackers comprisingplural illuminators a single image in which two reflections appear maybe sufficient to achieve the same result.) The choice of activecamera—and equivalently, the choice between imaging in the bright-pupilor dark-pupil mode—depends on the actual image quality obtained usingeach camera, as outlined above. The image quality metric, which may bedetermined by an external processing means, may take several qualityfactors into account, as will be further discussed in section IV below.

The arrangement 100 may be operable in a dual-camera mode, whichimproves the accuracy in finding the glint. Referring to the paper byGuestrin and Eizenmann's paper (see above), the extra informationobtained by the second camera is added as more rows added into equationsystem (18), which will be solved to give the centre of cornealcurvature c. The added rows will imply that the centre of cornealcurvature is determined with greater accuracy. In exceptionalcases—e.g., when the spatial configurations of the cameras isunfortunate—the addition of rows may actually lead to an increasedcondition number of the matrix; then the single-camera mode may betemporarily resumed.

To find the position of an eye, one may determine the location of anilluminator (the world coordinate of which is a priori known) relativeto the eye by considering its cornea-scleral reflection. When a singlecamera is used, the illuminator location X_(ill) may be computed as anaverage weighted by the intensities:

$X_{ill} = {\frac{\sum\limits_{i \in G}\; {{X_{i}I}\; N\; T_{i}}}{\sum\limits_{i \in G}{I\; N\; T_{i}}}.}$

Here G is a set of pixels in the image in which the glint is contained,INT_(i) is the intensity (after subtraction by the background intensity)of the ith pixel, and X_(i) is the world coordinate of a light sourcethat gives a corneo-scleral reflection in the ith pixel. However, whentwo cameras are used, the calculation can be refined as follows:

$X_{ill} = \frac{{\sum\limits_{i \in G}\; {{X_{i}I}\; N\; T_{i}}} + {\sum\limits_{i \in H}\; {{X_{i}I}\; N\; T_{i}}}}{{\sum\limits_{i \in G}{I\; N\; T_{i}}} + {\sum\limits_{i \in H}{I\; N\; T_{i}}}}$

where G is a set of pixels of the first camera's image containing theglint, and H is a set of pixels of the second camera's image containingthe glint. By linearity, assuming the two cameras are of identical type,the standard deviation of the estimation of X_(ill) decreases by afactor of up to 1/√{square root over (2)} if the dual-camera equation isused instead of the single-camera equation.

II. Eye Tracker Comprising Plural Reference Illuminators

FIG. 2 shows a combined camera and illuminator arrangement 200. Justlike the arrangement 100 of FIG. 1, it comprises first and secondcameras 210, 212. However, the present arrangement 200 is equipped withfour reference illuminators 220-226. A first reference illuminator 220of these is coaxial with the first camera 210 and a second referenceilluminator 222 is coaxial with the second camera 212. The cameras 210,212 are situated some distance apart, so that the first referenceilluminator 220 is non-coaxial with the second camera 212 and the secondreference illuminator 222 is non-coaxial with the first camera 210. Theleast distance at which non-coaxiality (the cease of the bright-pupileffect) occurs is dependent on the distance from the arrangement 200 tothe eye, the actual pupil diameter and various parameters that aresubject to individual variation. Typically, if the arrangement 200 isdesigned for measurements on eyes not further away than 1 m, then aseparation of the cameras by 100 mm may be considered sufficient. Thethird and fourth reference illuminators 224, 226, which are respectivelyarranged on the left and right side of the arrangement 200, are suitablyactive at large lateral gaze angles. This way, a centrally located glintcan be obtained also at these angles.

FIG. 3 depicts another embodiment of a combined camera, illuminator andvisual display arrangement 300. The reference illuminators and camerasof the arrangement 300 are provided around the edge of a screen surface340 for displaying graphical information. In contrast to thearrangements shown in FIG. 2, in which the reference illuminators arealigned one-dimensionally, the present arrangement 300 comprisesreference illuminators 320-338 having a two-dimensional configuration.Not only does this increase the range of gaze angles for which a centralcorneal glint can be achieved. It also enables assessments of mappingsinvolving angular deformations or differing horizontal and verticaldeformations. For instance, reflection in a surface having two differentradii of curvature, notably an elliptic surface, will deform a squareinto a rectangle or a parallelogram. By assessing the ratio of thehorizontal and vertical length scales under the reflection, informationrelating to the reflection point on the surface can be obtained. Bystudying how the different angles change under the reflection, it ispossible to estimate the skewness of the surface around the reflectionpoint.

III. PCCR Gaze Tracking Using an Aspherical Corneal Model

Gaze tracking using an aspherical cornea model, more particularly anellipsoidal cornea model, will now be outlined. FIG. 5 diagrammaticallydepicts the experimental situation. Reference illuminators 912, each ofwhich is independently activable, are provided in an object plane 910.The illuminators 912 are imaged as corneal reflections 926 in the cornea922 or sclera 924 of a person's eye 920. A camera 930, which ispreferably a digital imaging device, images the corneal reflections 926as image points 934. In a simplified model, as shown on the drawing, theimaging of the camera 930 is determined by a (rear) nodal point 932 andan image plane. For clarity, light rays are indicated from referenceilluminators 912 a, 912 b and 912 d only. The compound imaging processof the cornea 922 and the camera 930, which maps each referenceilluminator 912 to an image point 934, can be expressed by the followingmathematical relationship:

X′=[Proj·Refl_(T(E))](X),

Where Proj is a perspective projection (which in homogeneous coordinatesis a linear mapping) known through camera calibration;

E is an ellipsoid representing the corneal surface, known throughpersonal calibration of the test subject while focusing sample points;

T is a rigid transformation which reflects the actual position andorientation of the ellipsoid;

X is a coordinate vector for an illuminator known through thepredetermined illuminator arrangement; and

X′ is a coordinate vector for the camera image of the same illuminator.

The reflection map Refl_(T(E)) (which is determined by the assumptionsof rectilinear propagation of light and of equality between angles ofincidence and reflection; in computer-graphics terminology it is an‘environment map’) depends parametrically on T(E) which, in turn, is afunction of the actual position and orientation T of the cornea. WhenT(E) is found, such that

Proj⁻¹(X′)=Refl_(T(E))(X)

holds true (this equation is equivalent to the previous one), theposition and orientation of the eye are known, and the gaze vector canbe determined in a straightforward manner. The parameters specifying themappings Proj and Refl_(T(E)) can be estimated by considering pairs ofknown object and image points (X,X′), preferably the referenceilluminators and their images under reflection in the cornea. Once themappings are known, it is possible to find counterparts of object pointsin the image and vice versa; particularly, the location of the pupilcentre can be mapped to the image to provide an approximate gaze point.

A procedure of solving the gaze-detection problem will now be outlined;one of its advantages over gaze detection via a complete estimation ofthe mappings Proj and Refl_(T(E)) is that sufficient information forfinding the gaze-point may be obtained with fewer computations and lessinput data. The ellipsoid E used to model the cornea is more preciselygiven as a surface of revolution, with respect to the x axis, of thecurve

$\begin{matrix}{y^{2} = \left. {{2\; r_{0}x} - {p\; x^{2}}}\Leftrightarrow{\left( \frac{x - {r_{0}\text{/}p}}{r_{0}\text{/}p} \right)^{2} + \left( \frac{y}{r_{0}\text{/}\sqrt{p}} \right)} \right.} \\{{= 1},}\end{matrix}^{2}$

where p<1 (the ellipsoid is prolate), x is the dorso-ventral coordinateand y is the vertical coordinate. An ellipsoid having this shape isshown in FIG. 4, wherein the line AA′ represents the x axis and the ydirection is vertical on the drawing. In a three-dimensionaldescription, if a lateral coordinate z is included, E is defined by

${\left( \frac{x - {r_{0}\text{/}p}}{r_{0}\text{/}p} \right)^{2} + \left( \frac{y}{r_{0}\text{/}\sqrt{p}} \right)^{2} + \left( \frac{z}{r_{0}\text{/}\sqrt{p}} \right)^{2}} = 1.$

The arc SPS in FIG. 4 represents the sagittal radius of curvature, whichis given by

r _(S)(y)=√{square root over (r ₀ ²+(1−p)y ²)},

where y is the height coordinate of point P. The tangential radius ofcurvature, as measured on the arc TPT in the plane of the drawing, isdefined as

${r_{T}(y)} = {\frac{{r_{s}(y)}^{3}}{r_{0}^{2}}.}$

Points C_(S) and C_(T) are the respective centres of sagittal andtangential curvature at P. Because E is a surface of revolution, A:(0,0)is an umbilical point, at which both radii of curvature are equal to theminimal radius r₀. The described model is valid in the corneal portionof the eye, whereas the sclera has an approximately spherical shape.Typical values of the minimal radius and the eccentricity are r₀=7.8 mmand p=0.7, but vary between individual corneae. To achieve optimalaccuracy, these constants may be determined for each test subject in acalibration step prior to the gaze tracking. The calibration step mayalso include determining the distance from the pupil centre to thecorresponding centre C₀ of corneal curvature and the angular deviationbetween the visual and optic axes of the eye. It is noted that thespherical model is obtained as a special case by setting p=1 in theformulas above; as an immediate consequence hereof, the sagittal andtangential radii are equal.

The calculations may be carried out along the lines of the already citedarticle by Guestrin and Eizen-mann, however with certain modificationsto account for the aspherical cornea model. Following Guestrin andEizenmann, the locus of a reference illuminator 912 is denoted by L, thenodal point 932 of the camera is denoted by O and the image 934 of thecorneal reflection is denoted by U. Because each point P≠A on the corneahas two different radii of curvature in the ellipsoidal model, thearticle's co-planarity assumption of vectors {right arrow over(LO)},{right arrow over (OU)},{right arrow over (OC)}₀, by which notablyeach line of equation 15 follows, is no longer valid. In the case of anellipsoidal cornea model, separate equations are obtained for thetangential and sagittal components of the vectors. Separating {rightarrow over (OU)},{right arrow over (LO)} in sagittal and tangentialcomponents by orthogonal projection, as per

{right arrow over (OU)}={right arrow over (v_(S))}+{right arrow over (v_(t))},

{right arrow over (LO)}={right arrow over (w _(S))}+{right arrow over (w_(T))},

the following groups of co-planar vectors are obtained: {right arrowover (C_(S)P)}, {right arrow over (v_(s))}, {right arrow over (w_(s))}and {right arrow over (C_(T)P)}, {right arrow over (v_(T))}, {rightarrow over (w_(T))}. The calculations can then be continued in a mannersimilar to that disclosed in the article.

The inventors have found empirically that use of an ellipsoidal corneamodel leads to a significant increase in accuacy. It has even beenobserved that pupil-centre tracking is in some cases not necessary as asupplement to glint tracking, as practised hitherto in the art. Indeed,tracking of the cornea—apprehended as an ellipsoidal, rotationallyasymmetric surface—provides sufficient information (apart fromcalibration data such as the angular difference between the optic axisand the visual axis) that the orientation of the eye can be determined.Likewise, the process of calibrating certain parameters, notably theminimal radius of curvature and the eccentricity, can be simplified inso far as the test subject is not required to fix his or her eyes ontraining points. Such improvement of the calibration process isdependent on the correctness of the assumption that the optic axis ofthe eye coincides with the symmetry axis AA′. Further improvements maybe achieved by using a compound light pattern or a time-varying lightpattern for generating corneo-scleral glints.

IV. Method for Selecting a Combination of a Camera and a ReferenceIlluminator

With reference to FIG. 6, a preferred embodiment of a method forselecting a combination of an active camera and an active referenceilluminator will be described. The selection is made from a plurality ofreference illuminators adapted to illuminate at least one eye and aplurality of cameras adapted to image the eye or eyes with the aim ofselecting that combination which provides the most suitable conditionsfor gaze tracking of the eye(s).

In step a) of the method, an image quality metric is defined. The imagequality metric may be based on the quality factors indicated in TABLE 1below.

TABLE 1 Image quality factors NbrPupils The number of pupils detected bythe camera. Two detected pupils are preferred to one or none.GazeDetNoise If the test subject fixates a number of visible points in acalibration process, then parameters can be set to such values that theexpected divergence from the true point locations is zero. The gaze-detection noise after this process can be expressed as a statisticalmeasure (such as variance, standard deviation, maximal value etc.) ofthe divergence. A lower gaze-detection noise is preferred. PupilContrastThe difference in luminance of a region of the pupil and a region of theiris. Preferably, the regions are located centrally in the pupil and theiris, respectively, and the luminance values are averaged over theregions. A greater pupil contrast is preferred. IrisGradient Off-axisregions in a camera's field of view may have a lower (effective)resolution than central regions. The magnitude of the gradient at thepupil-iris boundary is taken as a measure of the resolution. A greatermagnitude of the gradient is preferred. Obstacles The pupil-irisboundary may be obscured by the presence of obstacles, such as eye-lashes, non-transparent parts of eye glasses, reflections from eye-glasslenses, glints, eyebrows, nose and the like. It is noted that the mostcentric glint may lie on the pupil-iris boundary and be detrimental tothe pupil finding; in such circumstances, it may be better to use theilluminator that gives the next most centric glint. The absence ofobstacles is preferred. SNR A signal-to-noise ratio can be defined bytaking PupilContrast (see above) as a measure of the signal intensityand the standard deviation at the centre of the pupil, which is anormally a monochrome region, as a measure of the noise. A highersignal-to-noise ratio is preferred.

Out of these quality factors, the inventors deem NbrPupils, GazeDetNoiseand PupilContrast to be the most important, whereas IrisGradient,Obstacles and SNR may be used as additional factors. The image qualityfactors may be combined into a total quality metric as per

Image  Quality = α₁NbrPupils + α₂GazeDetNoise + α₃PupilContrast + α₄IrisGradient + α₅Obstacles + α₆S N R,

where coefficients α₁,α₂, . . . ,α₆ are constants of appropriate signs.For instance, α₁ and α₂ should be of opposite signs, considering thepreferred values of the quantities. Since the image quality metric isonly used for establishing the relative quality of two images, there isno real need for an absolute calibration of the sub-metric. However, therelative weighting between sub-metrics, as reflected by the absolutevalues of the coefficients, should be chosen with some care to fit therequirements of the application.

The possible combinations of a camera and an illuminator fall into twogroups: combinations of two coaxial components and combinations of twonon-coaxial coponents. The combinations of coaxial components areadapted to image the eye(s) in the bright-pupil mode (a retinalretro-reflection complements the iris image), whereas the combinationsof non-coaxial components are adapted to image in the dark-pupil mode (acornea-scleral reflection complements the iris image). Step a) isfollowed by step b), in which either the bright-pupil or the dark-pupilimaging mode is selected. To this end, at least one image of the eye inthe dark-pupil mode and at least one in the bright-pupil mode areacquired. The comparison is more accurate if these at least two imagesare acquired closely in time, which also makes the selection processswifter. To maximize both these benefits, the images are acquiredsimultaneously if possible (that is, if only one bright-pupil image istaken) in this embodiment. Preferably, the images are acquiredsimultaneously. The image quality metric is evaluated for these images,and the imaging mode is selected in accordance with the highest value ofthe metric. If more than one image has been acquired in each mode, thenthe imaging mode of the image having the globally maximal quality metricis selected.

Upon completion of step b), the method proceeds to step c), wherein anactive camera is selected. The image quality metric is evaluated forimages acquired using combinations according to the selected imagingmode. Possibly, some images which were used in step b) may be usedagain. The winning quality metric value determines which camera isselected. In this step, just like in step b), the images for which theimage quality factor is assessed may be acquired while the device is inan evaluation mode.

It remains to select, in step d), an active reference illuminator to beused in combination with the selected active camera. An advantageous wayof finding the most suitable reference illuminator is as follows: usingan initially selected reference illuminator the corneo-scleralreflection is retrieved; the deviation from the pupil centre of thereflection is established; it is determined whether there is analternative reference illuminator which has such position in relation tothe initially selected illuminator (is located in a direction oppositethe deviation) that a more centric corneo-scleral reflection can beachieved; if such alternative reference illuminator is available, it isselected and the centricity of the corneo-scleral glint is reassessed;if no improvement to the centricity is achieved using the alternativereference illuminator, reversion to the initially selected referenceilluminator takes place. This procedure may be refined by taking intoaccount the magnitude of the reflection's deviation from the pupilcentre; for instance, a relatively small deviation may not motivate useof an alternative reference illuminator.

On completion of step d), a combination of an active referenceilluminator and an active camera has been selected. The centricity ofthe corneo-scleral reflection (step d)) is reassessed regularly, andthis may provoke a decision to switch to another reference illuminator.To avoid too frequent reassessment of the centricity, a delay D ofsuitable duration (which the skilled person should be able to determineby routine experimentation) is provided between repetitions of step d).The delay causes an intermittent repetition of step d). Choosing alonger delay D eases the computational load, but deteriorates theaccuracy of the eye tracker. It is also possible to provide a delay Dwith adaptive duration, which reflects empirically observed humaneye-movement patterns, such as saccadic movements. To maintain a highimage quality, the image quality metric is evaluated for the selectedcombination, in step e), at regular intervals (such as after everycompletion of step d) or after every 2^(nd), 5^(th), 10^(th) or 20^(th)completion). If the image quality is greater than or equal to apredetermined level, then the intermittent repetition of step d) isresumed. If however the image quality metric is below the predeterminedlevel although updating of the reference illuminator selection (step d))has been effected, then the camera selection is revised by repeatingsteps c) and d). Immediately after such repetition, in step e′), theimage quality metric is evaluated again. If the image quality metric isstill below the predetermined level, then the selection of imaging modeis revised by repeating steps b), c) and d); otherwise, the methodresumes the intermittent repetition of step d).

With reference to FIG. 7, an application of the described method to thearrangement 200 shown in FIG. 2 will now be outlined. The arrangement200 comprises first and second cameras 210, 212 and first, second, thirdand fourth reference illuminators 220, 222, 224 and 226. The combinationof camera 210 and illuminator 220 is coaxial, as is the combination ofcamera 212 and illuminator 222. The other six combinations arenon-coaxial. The decisions taken during execution of the method areillustrated in the form of a tree in FIG. 7. Nodes b1, c1, c2, d1, d2,d3 and d4 symbolise decision points; an arrow symbolises a decision toselect an imaging mode (on the top level), a camera (on the middlelevel) or an illuminator (on the lowest level); and the leaves symbolisea complete combination of an active camera and an illuminator, asindicated.

Assuming an image quality metric has been defined the first decisionpoint b1 is whether to use the bright-pupil (BP) or dark-pupil (DP)imaging mode. If the bright-pupil mode is chosen, the method moves todecision point c1, at which the most suitable of the first camera 210and the second camera 212 is selected. No more decision is taken if thefirst camera 210 is selected, for only the first illuminator 220 iscoaxial with the first camera 210, and likewise, a selection of thesecond camera 212 inevitably implies that the combination with thesecond illuminator 222 will be used. Hence, decision points d1 and d2are trivial. If instead the dark-pupil mode is selected (at decisionpoint b1), each choice of an active camera (at decision point c2) leadsto a choice of three possible reference illuminators (at each ofdecision points d3 and d4). When the method has reached one of theleaves in the decision tree, the initial selection of acamera-illuminator combination is complete.

The selection is updated by climbing one level up in the tree. As noted,the selection of a reference illuminator is trivial in the case ofbright-pupil imaging, but at decision point d3 for instance, there is achoice between the second, third and fourth illuminators 222, 224, 226.The second illuminator 222 is likely to give the most centric cornealreflection for tracking a central gaze direction, whereas the third andfourth illuminators 224, 226 are probably suitable for lateral gazedirections. The switching may be performed by a simple controlmechanism. If evaluation of the image quality metric reveals thatupdating of the active illuminator selection cannot provide sufficientimage quality, the middle decision level is resumed (backwards along thearrows of the decision tree) and possibly the top level as well, shouldthe image quality not have improved sufficiently.

V. Closing Remarks

While the invention has been illustrated and described in detail in thedrawings and foregoing description, such illustration and descriptionare to be considered illustrative or exemplary and not restrictive; theinvention is not limited to the disclosed embodiments. For example, thespatial arrangement of the reference illuminators can be varied as wellas their number, and the image quality metric can be adapted to thepreferences of the intended users of each particular embodiment.

Other variations to the disclosed embodiments can be understood andeffectuated by those skilled in the art in practicing the claimedinvention, from a study of the drawings, the disclosure, and theappended claims. In the claims, the word ‘comprising’ does not excludeother elements or steps, and the indefinite article ‘a’ or ‘an’ does notexclude a plurality. A single processor or other unit may fulfil thefunctions of several items received in the claims. The mere fact thatcertain measures are recited in mutually different dependent claims doesnot indicate that a combination of these measured cannot be used toadvantage. A computer program may be stored or distributed on a suitablemedium, such as an optical storage medium or a solid-state mediumsupplied together with or as part of other hardware, but may also bedistributed in other forms, such as via the Internet or other wired orwireless telecommunication systems. Any reference signs in the claimsshould not be construed as limiting the scope.

1. An eye tracker comprising: at least one reference illuminator forilluminating an eye; a plurality of cameras for imaging the eye, with atleast one camera being arranged coaxially with said at least onereference illuminator, and with at least one other camera being arrangednon-coaxially with said at least one reference illuminator; and acontroller configured to repeatedly evaluate an image quality metricbased on at least one image quality factor for images provided by saidat least one camera and said at least one other camera, and to selectsaid at least one camera or said at least one other camera to be activebased on the image quality metric.
 2. The eye tracker according to claim1, wherein said at least one reference illuminator comprises a pluralityof reference illuminators.
 3. The eye tracker according to claim 2,wherein each camera is associated with a coaxial and substantiallypoint-shaped illuminator of said at least plurality of referenceilluminators.
 4. The eye tracker according to claim 2, wherein saidcontroller is further configured to repeatedly select a combination ofone of said at least one camera and said at least one other camera to beactive, based on the image quality metric and a centric location of acorneo-scleral reflection.
 5. The eye tracker according to claim 1,wherein said controller is configured to select said at least one cameraand said at least one other camera so that the eye tracker is operate ina dual-camera mode.
 6. The eye tracker according to claim 5, whereinsaid plurality of cameras are separated by 70 mm or more.
 7. The eyetracker according to claim 5, wherein said controller is furtherconfigured to determine a location of a first Purkinje reflection basedon two images acquired simultaneously by different cameras in thedual-camera mode.
 8. The eye tracker according to claim 2, wherein saidcontroller is operable in an evaluation mode and activates said at leastone camera and said at least one other camera while sequentiallyscanning the plurality of reference illuminators.
 9. The eye trackeraccording to claim 1, wherein said at least one reference illuminator isconfigured to emit at least one of infrared light and near-infraredlight.
 10. The eye tracker according to claim 2, wherein said controlleris further configured to determine a gaze direction of the eye based onlocations of corneo-scleral reflections of said at least one referenceilluminator.
 11. The eye tracker according to claim 2, furthercomprising a processor configured to: define a mapping between acoordinate system in an object coordinate plane and a coordinate systemin a plane of an eye image based on locations in the eye image ofcorneo-scleral reflections of said plurality of reference illuminators;and determine, based on the mapping, a gaze point of the eye in theobject coordinate system; with the mapping comprising an ellipsoidalreflection mapping and a perspective projection.
 12. The eye trackeraccording to claim 1, wherein the at least one image quality factorcomprises at least one of a number of detected pupils, a gaze-detectionnoise, a pupil contrast, a gradient of pupil-iris boundary, a presenceof obstacles, and a signal-to-noise ratio.
 13. A method for selecting acombination of an active illuminator from a plurality of referenceilluminators for illuminating an eye and an active camera from aplurality of cameras for imaging the eye, with at least one combinationcomprising an illuminator and a camera which are coaxial, and with atleast one combination comprises an illuminator and a camera which arenon-coaxial, the method comprising: a) defining an image quality metricbased on at least one image quality factor; b) selecting an imaging modebased on values of the image quality metric for at least two eye images,with one of the eye images being obtained using a coaxial combinationand with the other eye image being obtained using a non-coaxialcombination; c) selecting an active camera based on values of the imagequality metric for eye images acquired using combinations according tothe selected imaging mode; and d) selecting an active referenceilluminator based on the centricity of a corneo-scleral reflection ofeach reference illuminator from the plurality of reference illuminators.14. The method according to claim 13, further comprising the sequentialsteps of: e) if the at least one image quality factor is below apredetermined level, performing steps c) and d) and then, if the atleast one image quality factor is still below the predetermined level,further performing steps b) and c); and f) performing step d).
 15. Themethod according to claim 13, wherein the at least one image qualityfactor comprises at least one of a number of detected pupils, agaze-detection noise, a pupil contrast, a gradient of pupil-irisboundary, a presence of obstacles, and a signal-to-noise ratio.
 16. Anon-transitory computer-readable medium having computer executableinstructions for causing an eye tracker to perform steps comprising:selecting a combination of an active illuminator from a plurality ofreference illuminators for illuminating an eye and an active camera froma plurality of cameras for imaging the eye, with at least onecombinetion comprising an illuminator and a camera which are coaxial,and with at least one combination comprises an illuminator and a camerawhich are non-coaxial, the selecting comprising a) defining an imagequality metric based on at least one image quality factor; b) selectingan imaging mode based on values of the image quality metric for at leasttwo eye images, with one of the eye images being obtained using acoaxial combination and with the other eye image being obtained using anon-coaxial combination; c) selecting an active camera based on valuesof the image quality metric for eye images acquired using combinationsaccording to the selected imaging mode; and d) selecting an activereference illuminator based on the centricity of a corneo-scleralreflection of each reference illuminator from the plurality of referenceilluminators.
 17. The non-transitory computer-readable medium accordingto claim 16, further comprising the sequential steps of: e) if the atleast one image quality factor is below a predetermined level,performing steps c) and d) and then, if the at least one image qualityfactor is still below the predetermined level, further performing stepsb) and c); and f) performing step d).
 18. The non-transitorycomputer-readable medium according to claim 16, wherein the at least oneimage quality factor comprises at least one of a number of detectedpupils, a gaze-detection noise, a pupil contrast, a gradient ofpupil-iris boundary, a presence of obstacles, and a signal-to-noiseratio.